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CHAPTER  1 


INTRODUCTION 

A  major  issue  in  the  development  and  acquisition  of  military- 
systems  is  cost,  whether  it  be  cost  to  develop,  cost  to  buy,  cost  to 
support,  or  cost  to  maintain  (5:6).  Estimating  costs  of  proposed 
systems  takes  on  increasing  significance  as  the  Department  of 
Defense  (DOD)  has  consistently  underestimated  the  cost  of  major 
weapon  systems  (8:105).  Accuracy  and  realism  of  weapon  system 
cost  estimates  are  prerequisites  for  the  efficient  functioning  of  the 
Defense  Systems  Acquisition  Review  Council  (DSARC),  an  advisory 
body  to  the  Secretary  of  Defense  on  all  major  defense  systems, 
acquisition  programs  and  related  policy.  At  each  major  program 
decision  point,  all  aspects  of  the  program  are  reviewed  by  the 
DSARC.  Accordingly,  emphasis  on  improving  service  cost  estimates 
continues  to  receive  high  DOD  priority.  In  this  effort  the  Secretary 
of  Defense  Cost  Analysis  Improvement  Group  (CAIG)  was  formed  to 
exercise  the  dual  function  of  reviewing  service  cost  estimates  for 
the  DSARC  and  fostering  defense-wide  improvements  in  military 
cost  analysis  capabilities  (14:7). 

Decisions  to  undertake  specific  development  projects 


depend  critically  upon  the  estimates  of  development  and  production 
costs  and  to  the  extent  that  cost  estimates  for  planning  are  wrong, 
the  decisions  to  undertake  specific  projects  are  questionable. 

Further,  once  a  decision  is  made  to  start  a  specific  development 
project,  cost  estimates  play  an  important  role  in  selecting  con¬ 
tractors  and  monitoring  contractor  performance.  Being  able  to 
identify  overly  optimistic  cost  estimates  and  to  provide  incentives 
to  control  cost  during  development  and  production  are  important 
elements  of  the  government  acquisition  process.  Both  depend  heavily 
on  the  quality  of  cost  information  (15:7). 

In  the  development  phase  of  electronic  systems  the  methods 
used  to  predict  test  and  evaluation  (T&E)  or  more  specifically  flight 
test  costs  need  to  be  constantly  updated  and  improved.  During  the 
test  program  formulation  process,  program  managers  must  consider 
many  budgetary  aspects  in  establishing  funding  for  their  respective 
T&E  efforts.  When  a  program  office  requires  support  from  a  test 
center,  a  Program  Introduction  Document  (PID)  is  submitted  to  the 
Responsible  Test  Organization  (RTO).  The  PID  is  a  scope  setting 
document  describing  the  program,  identifying  the  known  test  support 
requirements,  forecasting  events,  and  specifically  spelling  out  the 
program  requirements.  It  is  used  by  the  RTO  for  basic  support 
planning  and  as  a  basis  for  determining  test  center  resources  or  new 
capabilities  requirements  (17:86). 


Upon  receipt  of  the  PID  the  RTO  begins  preparing  a  State¬ 
ment  of  Capability  (SOC).  This  document  provides  a  preliminary 
statement  of  the  support  the  user  may  expect  and  the  resources 
needed  to  satisfy  the  program  requirements.  It  delineates  the 
responsibilities  between  the  RTO  and  the  user,  including  the  pro¬ 
visions  of  resources  and  the  listing  of  program  constraints  (16:86). 
Along  with  this  procedure  the  program  office  develops  a  Test  and 
Evaluation  Master  Plan  (TEMP).  This  document  presents  the  over¬ 
all  system  description,  operations  and  maintenance  concepts, 
program  objectives,  areas  of  risk,  required  test  assets,  and  pro¬ 
gram  responsibilities  (2:11).  These  reports  are  the  basis  for  budget 
submissions  for  both  the  test  center  and  the  program  offices. 

Direct  cost  funding  requires  that  test  peculiar  costs  asso¬ 
ciated  with  a  particular  test  program  be  reimbursed  by  the  program 
office  to  the  designated  test  center.  The  cost  associated  with  special 
instrumentation,  operation  of  test  resources  (e.g.,  test  aircraft, 
range  operation),  overtime,  civilian  salaries,  travel  and  per  diem, 
transportation  and  communication  equipment  are  examples  of  the 
types  of  cost  that  may  be  charged  a  user  (3:p.  10-3).  Since  the  pro¬ 
gram  office  will  be  paying  for  the  work  performed,  it  needs  to  obtain 
an  estimate  from  the  test  center  as  to  how  much  the  work  will  cost. 
This  means  defining  the  test  requirements  and  getting  together  with 
the  test  center  personnel  some  two  years  in  advance  of  the  actual 
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work.  The  long  lead  time  necessary  for  budgeting  procedures  of 
direct  cost  funding  makes  it  difficult  for  program  offices  to  specif¬ 
ically  define  the  requirements  of  the  test  (17:86). 


Problem  Statement 

Estimating  flight  test  costs  for  electronic  systems  using 
current  procedures  (detailed  estimating),  generally  requires  a  well 
defined  program.  During  early  phases  of  Research,  Development, 
Test  and  Evaluation  (RDT&E),  this  level  of  detail  is  often  not  avail¬ 
able.  In  many  cases  the  program  office  has  been  unable  to  sufficient¬ 
ly  define  test  requirements  two  years  in  advance  as  mandated  by  the 
budget  process.  Even  though  sufficient  data  is  not  available  for  a 
detailed  estimate  some  type  of  cost  figures  must  be  included  in  the 
budget.  If  the  user  is  truly  unable  to  determine  his  long  lead  time 
requirements,  the  cost  estimate  degenerates  to  a  "best  guess"  or 
expert  opinion  approach  (5;  17:34).  Currently,  there  is  not  a  cost 
prediction  model  to  estimate  flight  test  direct  costs  for  electronic 
systems  to  aid  in  budget  preparations. 

Background 

The  scope  of  a  test  program  covers  the  development  process 
from  the  conceptual  phase  through  the  production  decision  and  beyond. 
The  initial  phase  of  testing,  defined  as  Development  Test  and 


4 


Evaluation  (DT&E),  is  conducted  to  demonstrate  that  the  engineering 
design  and  development  process  is  complete,  design  risk  has  been 
minimized,  and  the  system  meets  engineering  and  operational 
specifications.  The  second  area  of  testing  is  Operational  Test  and 
Evaluation  (OT&E),  which  is  normally  conducted  in  two  phases. 

Initial  Operational  Test  and  Evaluation  (IOT&E)  is  conducted  to 
provide,  before  the  first  major  production  decision,  a  valid  estimate 
of  expected  system  operational  effectiveness  and  suitability.  Follow- 
on  Operational  Test  and  Evaluation  (FOT&E)  is  conducted  as  neces¬ 
sary  during  and  after  the  production  period  to  refine  the  estimates 
made  during  the  IOT&E.  The  system  is  reevaluated  to  ensure  that  it 
continues  to  meet  operational  needs  and  retains  its  effectiveness  in  a 
new  environment  or  against  a  new  threat  (3:p.  10-3). 

Flight  testing  of  an  aircraft  or  other  systems  by  actual  flight 
is  planned  to  achieve  specific  test  objectives  and  gain  operational 
information  (21:298).  Fundamental  purposes  of  flight  testing  are: 

1.  To  determine  the  actual  characteristics  of  the  machine 
(as  contrasted  to  the  computer  or  predicted  results). 

2.  To  provide  developmental  information. 

3.  To  provide  research  information  (4:11). 

The  major  problems  encountered  in  flight  testing  work  include: 

l.  Measurement  of  the  actual  values  obtained  during  the 


2.  Determination  of  what  the  measured  values  would  have 
been  under  some  arbitrary-  set  of  standard  conditions. 

3.  Designing  of  the  test  program  to  provide  the  desired 
results  for  the  least  cost  in  time  and  money-  within  the  limitations  of 
available  manpower  and  equipment  (4:111). 

As  stated  earlier,  flight  tests  are  used  to  validate  predicted 
results.  As  an  example,  the  high  cost  of  air  launched  guided  missiles 
makes  it  virtually-  impossible  to  demonstrate  release  characteristics 
at  all  combinations  of  aircraft  angle -of- attack,  release  mode,  normal 
load  factor,  flight  path  angle  and  carriage  locations  throughout  the 
speed/altitude  release  envelope.  Current  practices  call  for  compari¬ 
son  of  wind  tunnel  based  predicted  trajectories  with  actual  in-flight 
separation  characteristics  at  predetermined  buildup  and  demonstra¬ 
tion  flight  conditions.  If  the  analytical  techniques  are  considered 
validated  and  satisfactory  correlation  is  obtained,  flight  characteris¬ 
tics  can  be  documented  by  less  expensive  computer  analysis.  If, 
however,  correlation  is  not  satisfactory  it  then  becomes  necessary  to 
perform  sufficient  drops  for  comparison  with  the  analysis  until  an 
acceptable  confidence  level  is  achieved  through  the  update  and  refine¬ 
ment  process  based  on  actual  in-flight  data.  The  success  of  the 
entire  procedure  is  contingent  on  the  accuracy  and  reliability  of  both 
the  aerodynamic  prediction  technique  and  the  flight  test  data  reduction 
process  to  minimize  the  number  of  drops  and  flights  required  in 
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establishing  safe  and  operational  clearance  envelopes  (18:p.  1-2). 

The  introduction  of  electronic  data  processing  to  general 
flight  test  operations  has  fostered  the  development  of  new  methods  of 
analysis  as  well  as  simply  mechanizing  old  accepted  methods.  Com¬ 
puter  oriented  data  systems  of  various  size  and  complexity  have 
become  a  necessary  tool  for  collection  and  verification  of  test 
results.  For  the  Rockwell  B-1B  test  program  performance,  data 
handling  and  processing  start  with  over  90  million  data  points  per 
flight.  There  are  more  than  1900  parameters  recorded  at  least  four 
times  a  second,  many  of  which  are  recorded  sixty-four  times  a 
second.  Multiplying  this  by  an  average  flight  time  of  seven  to  eight 
hours  results  in  a  tremendous  mass  of  raw  data  (12:p.  14-2).  Com¬ 
puter  programs  are  designed  to  identify  aircraft  stability  and  control 
from  dynamic  flight  test  data.  Modern  high  speed  digital  computers 
are  employed  which  reduce  cost  and  flow  time  of  flight  data  analysis. 

The  major  Air  Force  testing  centers  are  faced  with  the 
challenge  of  developing  sophisticated  analytical  techniques  and  safe, 
productive  flight  test  methods  for  today's  complex  weapon  systems. 
After  the  requirements  for  flight  test  have  been  set,  the  flight  test 
engineer  will  have  to  organize  the  testing.  The  engineer  defines  the 
test  to  be  executed,  how  many  aircraft  will  be  used  for  the  test,  the 
time  schedule,  the  sequence  of  the  tests,  the  test  procedure  for  each 
specific  test,  and  the  location  where  the  test  will  take  place  (1:7). 


Air  Force  flight  testing  is  done  at  the  following  bases: 

Edwards  AFB,  California 
Eglin  AFB ,  Florida 
Hill  AFB,  Utah 
Patrick  AFB,  Florida 
Vandenberg  AFB,  California 

The  cost  of  a  flight  test  is  composed  of  several  items.  The 
relative  amounts  of  each  of  these  items  depends  on  the  amount  of 
existing  equipment  and  the  type  of  testing  being  done.  The  operation 
of  the  aircraft  during  the  flight  test  will  constitute  a  significant  part 
of  the  test  budget.  Experienced  test  pilots,  flight  engineers,  ground 
crew,  and  support  are  required  for  test  flights.  Other  costs  include: 

1.  Cost  of  the  flight  test  definition  and  planning. 

2.  Flight  test  instrumentation  composition  cost. 

3.  Data  processing  hardware  composition  cost. 

4.  Flight  test  instrumentation  installation  and  checkout  cost. 

5.  Data  processing  software  composition  and  checkout  cost. 

6.  Flight  test  instrumentation  operation  cost. 

7.  Data  processing  cost. 

The  task  of  the  RTO  is  to  design  the  equipment,  software  and  pro¬ 
cedures  to  execute  the  flight  test  at  minimal  cost  (1:12). 
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Literature  Review 

Previous  studies  done  in  this  area  concentrated  on  techni¬ 
ques  to  predict  the  flight  test  costs  for  new  airframes  and  not  for 
new  electronic  systems  installed  on  an  already  existing  aircraft. 

The  Air  Force  Flight  Test  Center  (AFFTC),  at  Edwards  AFB,  is 
concerned  with  estimating  cost  of  flight  tests  and  they  have  developed 
a  technique  for  their  own  use.  This  approach  separates  costs  into 
two  components:  variable  costs ,  or  costs  allocated  directly  to  flying 
hours,  and  fixed  costs,  or  the  sustaining  manpower  and  other  ser¬ 
vices  necessary  to  support  flight  testing  other  than  direct  flying 
efforts.  To  use  this  procedure  requires  an  estimate  of  test  aircraft 
flying  hours  and  total  flight  tests  duration  in  months.  The  flight  test 
estimate  is  the  combination  of  program  duration  in  months  times  the 
average  monthly  program  fixed  costs  plus  the  prime  mission  aircraft 
flying  hours  times  the  variable  cost  per  flying  hour  (7:3).  This  esti¬ 
mate  relies  on  an  accurate  prediction  of  the  necessary  resources 
(monthly  program  costs)  needed  in  the  test  which  may  be  difficult  for 
new  airframes. 

Another  approach,  developed  by  Rand  Corporation,  is  to  use 
the  parametric  method.  Parametric  cost  analysis  uses  estimating 
relationships  developed  from  historical  program  costs  as  a  function 
of  variables  such  as  physical  and  performance  characteristics  (21: 
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511).  The  Cost  Estimating  Relationship  (CER)  included  data  from 
fifteen  fighter,  eight  bomber  and  transport,  and  seven  attack  air¬ 
craft.  The  best  relationship  included  empty  weight,  speed,  and 
number  of  test  aircraft  as  the  explanatory  variables .  This  study 
concentrated  on  testing  of  aircraft  airframes  and  did  not  include  the 
testing  of  other  systems  (11:92-96). 

Another  parametric  study  was  done  by  the  Directorate  of 
Cost  Analysis,  Aeronautical  Systems  Division  (ASD).  This  study 
computed  dollars  per  flying  hour  by  dividing  test  aircraft  flying  hours 
into  total  flight  test  cost.  To  develop  a  CER,  the  parameters  con¬ 
sidered  were  gross  take-off  weight,  average  sortie  length,  test 
aircraft  flying  hours,  and  maximum  design  speed.  The  aircraft 
involved  in  the  study  were  the  F-15,  F-16,  A- 10,  and  the  B-l.  Only 
gross  take.-off  weight  was  included  in  the  equation  as  use  of  multiple 
regression  techniques  were  precluded  as  a  result  of  the  small  sample 
size  (6:1-3).  Again,  this  study  dealt  with  airframe  testing. 

Lockheed- Georgia  Company  also  used  a  parametric  technique 
to  estimate  flight  test  costs  for  the  Advanced  STOL  Transport  Air¬ 
craft.  Data  from  the  C-130,  C-140,  C-141,  YC-130  and  C-5,  along 
with  available  data  from  a  number  of  other  aircraft  developed  by 
other  manufacturers  was  included  in  the  data  base.  Explanatory 
variables  included  empty  weight,  maximum  cruise  speed  and  number 
of  test  aircraft.  This  study  concentrated  on  transport  airframe 


testing  ( 13  sAPPENDIX  II) 


Scope  and  Limitations 

This  project  will  deal  with  developing  a  cost  prediction 
model  for  flight  testing  electronic  systems.  As  seen  by  the  liter¬ 
ature  review  results ,  no  efforts  to  estimate  avionic  or  electronic 
system  flight  tests  have  been  documented.  Predicting  flight  testing 
requirements  and  costs  is  more  difficult  in  this  area  because  there 
are  large  numbers  of  multi-varied  electronic  systems/subsystems 
with  non-major  program  status.  These  systems  require  dissimilar 
aircraft  (e.g.t  F-16,  F-15,  A- 10,  F-4)  to  support  required  flight 
test  environments. 


Research  Objective 

The  objective  of  this  research  is  to  develop  a  cost  prediction 
model  for  electronic  systems  flight  test  direct  cost  or  cost  change  to 
the  user.  This  model  could  then  be  used  by  program  office  personnel 
as  an  aid  in  budget  preparations . 

Research  Questions 

Answers  to  the  following  questions  will  provide  the  means  to 
fulfill  the  research  objective. 


1.  What  costs  are  included  in  flight  testing  of  different 


systems? 

2.  What  are  the  significant  cost  drivers  in  developing  an 
estimation  model  for  flight  test  costs. 

3.  How  can  these  cost  drivers  be  best  related  to  accurately 
predict  future  flight  test  costs. 


CHAPTER  2 


METHODOLOGY 

General  Overview 


The  objective  of  this  research  was  to  develop  a  cost  predic¬ 
tion  model  for  electronic  system  flight  test  costs.  This  model  was  a 
cost  estimating  relationship  (CER)  describing  a  numerical  relation¬ 
ship  which  is  useful  in  computing  estimated  costs.  These  estimates 
will  be  made  during  early  phases  of  the  program  when  the  details  are 
not  well  defined.  Due  to  the  nature  of  the  problem  a  parametric  cost 
model  resulting  from  multiple  regression  techniques  will  be  used.  A 
simulation  was  not  done  because  the  level  of  detail  in  the  data  was  not 
available.  A  recent  study  within  the  DOD  discussed  the  merits  of  the 
parametric  approach.  During  the  early  phases  of  the  acquisition 
process,  only  limited  information  is  available,  so  considerable 
uncertainty  surrounds  both  this  information  and  whatever  planning 
data  is  available  on  how  the  new  system  will  be  developed  and  pro¬ 
duced.  Nonetheless,  cost  estimates  must  be  made.  Both  the  fact  of 
limited  and  uncertain  information  on  which  to  base  estimates,  and  the 
use  to  be  made  of  these  estimates,  strongly  suggests  the  employment 
of  parametric  estimating  procedures  (9:2). 
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For  most  new  systems  the  parametric  approach  is  the  best 
method  that  can  be  used  to  make  an  estimate  from  the  limited  infor¬ 
mation  available  during  concept  formulation,  i.e.,  when  only  mission 
and  performance  requirements  are  defined.  Furthermore,  para¬ 
metric  methods  provide  the  analyst  with  an  inexpensive  means  of 
examining  the  impact  on  cost  of  a  variety  of  changes  of  system  per¬ 
formance  requirements.  In  short,  the  parametric  approach  is  based 
upon  aggregate  relationships  between  costs  and  the  physical  and  per¬ 
formance  characteristics  of  the  system  under  study.  These  relation¬ 
ships  are  derived  from  related  historical  data,  following  the  princi¬ 
ples  of  statistical  inference  (9:2).  Other,  less  desirable,  cost  esti¬ 
mating  methods  useable  during  the  same  time  period  are  those  called 
expert  opinion  and  analogy. 

The  parametric  method  has  limitations  that  must  be  consider¬ 
ed.  To  be  fully  effective,  it  requires  an  extensive  base  of  past  cost 
and  performance  data.  Its  use  implies  that  the  relationship  that 
existed  in  the  sample  on  which  the  estimating  equations  were  based, 
will  continue  to  exist  in  the  future.  Extrapolations  which  involve 
systems  with  a  large  advance  in  the  state-of-the-art  become  more 
difficult  as  the  new  program  differs  more  and  more  from  the  technol¬ 
ogy  which  existed  at  the  time  the  sample  programs  were  procured. 

The  technique  can  only  be  used  as  a  base  estimate  to  which  adjust¬ 
ments  are  made  to  allow  for  the  non- applicability  of  past  experience 
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Data  Collection 


Variables  to  be  Considered 

The  parametric  model  development  was  centered  on  the 
characteristics  of  the  test  effort.  The  dependent  variable  used  in 
this  study  is  direct  cost  of  flight  test  (those  costs  the  SPO  must 
reimburse  to  the  RTO)  measured  in  total  program  dollars.  The 
independent  variables  consisted  of  potential  cost  drivers.  Estimators 
are  continuously  searching  for  combinations  of  aircraft  character¬ 
istics  that  will  provide  consistently  reliable  estimates  and  be  logically 
related  to  cost.  The  potential  cause  and  effect  relationship  between 
each  potential  cost  driver  and  the  dependent  variable  was  evaluated  to 
determine  a  logical  link.  Only  then  can  a  potential  cost  driver  be 
considered  as  a  candidate  for  inclusion  in  the  model.  This  process 
of  identifying  the  determinants  of  flight  test  costs  for  electronic  items 
relied  heavily  on  the  knowledge  of  people  in  the  field.  Some  cost 
drivers  believed  to  effect  the  cost  of  flight  test  include  number  of 
sorties,  test  aircraft  flight  hours,  length  of  program,  and  the  type  of 
aircraft  (7:4).  These  performance  characteristics  deal  with  the  test 
design  and  not  characteristics  of  the  electronics  system  being  tested. 

A  restriction  in  the  process  of  identifying  cost  determining 
parameters  is  that  the  parameters  must  be  available.  There  are 


two  aspects  of  this  availability: 

1.  The  parameter  should  be  one  which  is  .specified  early  in 
the  life  cycle  and  does  not  require  completion  of  the  design  of  the 
test  before  it  is  known. 

2.  The  parameter  should  be  easily  and  unambiguously 
measurable  (9:20). 

Prior  to  incorporating  data  within  the  postulated  estimating  relation¬ 
ship,  it  must  be  consistently  defined.  Aside  from  uniform  definitions 


of  just  what  costs  are  included  within  the  recorded  expenditures, 
several  required  considerations  ensure  that  costs  are  displayed  in 
consistent  and  comparable  terms.  These  include  the  effects  of  (1) 
year-to-year  price  level  changes,  (2)  productivity  or  manufacturing 
technology  changes,  and  (3)  ,rlearning"  or  cost  quantity  effects  upon 


production  costs  (9:26). 


Data  Search 


Data  on  the  individual  systems  flight  tested  was  gathered 
from  the  individual  program  offices  located  at  Wright-Patterson  Air 
Force  Base.  Data  was  also  collected  from  the  3246th  Test  Wing 
located  at  Eglin  Air  Force  Base  as  most  electronic  system  testing 


is  done  there, 


Developing  the  Model 


A  multiple  regression  was  formulated  using  the  Statistical 
Package  for  the  Social  Sciences  (SPSS).  All  the  variables  were  forced 
into  the  equation  to  see  their  relative  effect  on  the  model.  The  least 
squares  method  of  regression  analysis  provides  the  best  unbiased 
estimator  of  the  dependent  variable,  therefore,  this  method  was  used. 
The  general  form  of  the  multiple  regression  model  is: 

Y  =  B0  +  BjXj  +  B2X2  +  B3X3  +  .  .  .  B^jXh.j  +  E 

where: 

Y  is  the  dependent  variable  representing  the  flight  test  costs. 
Xj's  are  the  independent  variables  representing  the  various 
flight  characteristics. 

Bj's  are  the  unknown  parameters  to  be  estimated  by  the 
analysis . 

E  is  the  error  term. 

The  specific  independent  variables  used  in  the  model  must  be  logi¬ 
cally  related  to  cost  and  this  logic  must  remain  consistent  throughout 
the  model  development. 

Solving  or  Manipulating  the  Model 

The  data  was  supplied  to  the  stepwise  program  in  order  of 
highest  range  of  values  to  the  smallest.  The  model  formulation 
explored  quadratic  terms  and  possible  interaction  terms  of  potential 
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independent  variables.  Characteristics  which  prove  not  to  be  statis¬ 
tically  significant  will  be  judged  to  determine  if  their  expected  effect 
on  cost  is  sufficient  to  remain  in  the  model.  The  resulting  equations 
were  evaluated  in  order  to  pick  the  best  predictor. 

Evaluating  the  Model 


In  evaluating  the  model,  the  relationship  between  the  inde¬ 
pendent  and  dependent  variables  had  to  remain  logically  consistent. 
If  an  independent  variable  was  expected  to  increase  cost  but  it  had 
a  negative  coefficient  causing  cost  to  decrease,  it  was  analyzed  to 
determine  why  it  was  behaving  this  way.  If  explanation  for  this 
behavior  is  not  possible  the  variable  was  excluded. 

Various  statistical  tests  were  performed  to  determine  the 
degree  to  which  the  independent  variables  predicted  the  dependent 
variable  accurately.  These  tests  are  discussed  separately. 


Coefficient  of  Determination  (RH 

R  measures  the  proportion  or  percentage  of  the  variation 
in  the  dependent  variable  explained  by  the  regression  model.  It  is 
defined  as  the  ratio  (10:525): 

Variation  Explained  _  c0e££iCien,t  of  Determination 
Total  Variation 

2  2 

Its  limits  are  0  IR  I  1.  An  R  of  one  means  a  perfect  fit,  whereas 
2 

an  R  of  zero  means  no  relationship  between  the  dependent  variable 
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and  the  explanatory  variables  (9:49).  From  talking  to  experienced 
analysts  in  this  field  an  acceptable  level  of  R  will  be  defined  as  .80 

(19). 

F-Test 

The  F-test  assesses  the  overall  statistical  significance  of 

the  model.  It  is  calculated  as  the  ratio  of  the  explained  variance  to 

the  unexplained  variance. 

Variation  Explained/k  _  p 
Variation  Unexplained /n-k-1 

where  k  is  the  number  of  independent  variables  and  n  is  the  number  of 
observations  in  the  sample  (10:548).  If  the  calculated  value  exceeds 
the  critical  value  (table  value)  at  the  80  percent  level  of  confidence 
the  model  will  be  considered  statistically  significant. 


t-Test 

The  t-test  measures  the  statistical  significance  of  each 
individual  coefficient.  It  is  measured  as  the  ratio  of  the  absolute 
value  of  the  coefficient  to  its  standard  error: 

t  -  lbl  -  ol 
'  S 

When  the  calculated  value  of  t  exceeds  the  critical  value  (table)  then 
the  variable  will  be  considered  statistically  significant  (10:550). 
Values  will  be  compared  at  the  .80  level  of  confidence. 


Additional  Test 


In  addition  to  using  statistical  techniques  to  evaluate  the 
model,  a  relative  error  calculation  will  be  made.  This  calculation 
examined  the  model  accuracy  by  comparing  the  prediction  error  as 
a  percentage  of  the  actual  flight  test  costs  for  each  historical  data 
point  used  in  generating  the  model.  This  relative  error  was  calcu¬ 
lated  based  on  the  following  formula: 

Actual  Test  Costs  -  Predicted  Test  Costs  ^  jqq 
Actual  Test  Costs 

Although  no  specific  level  will  be  defined,  this  calculation  can  help 
the  decision  maker  assessing  the  risk  in  accepting  the  estimate  by 
providing  a  more  intuitive  measure  of  the  errors. 


CHAPTER  3 


DATA  BASE  COLLECTION 

Sources  of  Data 

The  data  collection  process  began  at  the  ASD  Reconnaissance 
and  Electronic  Warfare  (RW)  System  Program  Office  (SPO)  to  deter¬ 
mine  what  programs  to  include  in  (he  data  base.  An  interview  with 
Mr.  John  Alien,  Director  of  Program  Control,  outlined  the  general 
type  of  testing  that  could  be  representative  of  electronic  system 
testing  as  well  as  some  specific  test  programs.  These  general 
areas  included  external  systems  or  pods  attached  to  the  aircraft, 
internal  systems,  and  dispensers  such  as  a  flare  that  is  released 
from  the  aircraft.  He  also  provided  points  of  contact  for  the  specific 
tests  that  were  identified. 

These  contacts  in  Program  Control  were  mainly  concerned 
with  accomplishing  test  objectives  and  whether  the  total  test  program 
was  within  the  budget*  Most  of  the  specific  data  easily  available  was 
the  schedule  or  estimated  effort  that  was  needed.  As  an  example, 
the  AN/ALQ-131  Receiver/Processor,  which  is  a  pod  attached  to  the 
F-4,  had  thirty  sorties  scheduled  (figure  provided  by  the  SPO)  and  in 
actuality  only  twenty-four  flights  were  used.  Actual  information  is 


available  at  the  SPO  but  this  data  originates  from  the  RTO.  There¬ 
fore,  to  easily  access  the  source  of  actual  data  and  to  expand  the 
potential  data  base  to  systems  not  specifically  supported  by  the  RW 
SPO,  the  RTO  became  a  source  of  data.  Most  electronic  system 
tests  are  conducted  by  the  3246th  Test  Wing  located  at  Eglin  Air 
Force  Base. 

The  necessary  data  was  collected  from  two  different  areas  at 
Eglin.  The  actual  cost  of  testing  the  specific  system  was  obtained 
from  their  Job  Order  Cost  Accounting  System  (JOCAS).  With  this 
system  program  specific  costs  are  collected,  documented,  and 
billed  to  the  appropriate  program.  Each  system  has  a  designated 
Job  Order  Number  (JON)  consisting  of  eight  digits.  The  first  four 
digits  are  either  the  basic  Air  Force  Systems  Command  (AFSC) 
system/project  number  or  the  appropriate  code  from  AFSC  Regula¬ 
tion  27-5  or  Armament  Division  (AD)  Regulation  80-1.  Of  the 
remaining  numbers,  digit  five  denotes  the  financial  manager,  digit 
six  denotes  the  customer  and  digits  seven  and  eight  provide  the 
sequence  of  the  test. 

Each  time  a  person  works  on  a  project  or  operates  a  system 
in  support  of  that  project  it  is  documented  by  the  JON  to  allow  for 
specific  billing.  These  cost  inputs  are  collected  and  grouped  into 
the  following  categories  on  the  billing  sheet: 


AIRCRAFT  AND  LOGISTICS 
COMPUTER  SUPPORT 
ENGINEERING  SUPPORT 
RANGE  OPERATION 
RANGE  SUPPORT 
TEST  MANAGEMENT 
TEST  REQUIREMENTS  SUPPORT 
AIRCREW  SUPPORT 
AIRBORNE  SUPPORT 
PHOTO  GRAPHICS 
ELECTRONICS 
MISCELLANEOUS 

For  programs  involving  foreign  government  testing  there  are 
lines  for  military  support,  civil  engineering  support,  and  overhead. 
These  costs  are  reimbursable  to  the  test  organization  for  non 
United  States  testing.  In  order  for  foreign  government  testing  to  be 
included  in  the  data  base,  the  costs  listed  for  military  support,  civil 
engineering  support,  and  overhead  were  subtracted  from  the  total 
test  direct  costs.  By  doing  this,  the  direct  cost  for  all  programs  in 
the  data  base  contained  the  elements  listed  on  a  United  States  test 
billing  sheet  and  could  be  easily  compared.  The  systems  affected 
were  PEACE  SUN,  PEACE  FOX,  and  PEACE  MARBLE. 

The  second  source  of  data  was  AD's  Management  Information 
System  (MIS)  which  contains  data  for  past  and  present  testing.  The 
number  of  flying  hours  for  each  type  of  aircraft,  duration  of  the  test 
in  months,  and  the  number  of  sorties  required  to  complete  the  testing 
were  obtained.  For  systems  that  used  multiple  aircraft  in  the  testing. 


the  dominant  (by  number  of  sorties)  aircraft  was  selected  to  represent 
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the  overall  system.  As  an  example,  the  ALR-56  restructuring  test 
included  sixty-two  flights  with  an  F-15,  two  flights  with  an  F-4C, 
two  flights  with  a  T-38,  and  one  flight  with  an  F-16.  In  this  example 
it  is  easy  to  determine  that  the  F-15  is  dominant.  In  all  the  systems 
there  was  a  distinct  principle  aircraft  in  the  test. 


Selection  of  Data  Points 


Systems  considered  for  inclusion  in  the  data  base  were  tested 
by  the  Electronic  Warfare  (TZW),  Electronic  System  (TZE),  and  Test 
Requirements  and  Programming  (TZP)  Divisions  of  AD  as  broken  out 
by  their  organizational  chart  (Appendix  A) .  More  specifically,  in  the 
electronic  warfare  area  systems  were  selected  from  both  the  attack 


and  defense  subdivisions.  In  the  electronic  systems  area  the  air¬ 
borne  systems  were  included  and  in  the  requirements  and  program¬ 


ming  area  the  subdivision  electronics  were  included. 

The  following  definitions  apply  to  tests  conducted  at  AD: 
Planning  Phase  -  The  planning  phase  begins  upon  acceptance 


of  an  effort  for  planning  purposes.  At  this  time  a  letter  authorizing 
the  planning  JON  and  identifying  the  eventual  direct  JON  is  trans¬ 
mitted  to  all  AD  agencies  involved.  During  the  planning  phase  a  test 
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directive  will  be  prepared  as  appropriate.  During  this  phase  AD 


agencies  are  authorized  to  only  charge  man-hours  to  the  JON.  Test 
planning  phase  ends  with  the  initiation  of  the  active  phase. 
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Active  Phase  -  The  active  phase  begins  when  the  scheduling 
of  the  test  effort  is  authorized.  During  this  phase  expenditure  of 
resources  is  authorized.  After  each  flight  a  report  is  completed 
describing  the  results  of  the  mission.  The  active  phase  ends  when 
the  last  test  mission  is  complete. 

Reporting  Phase  -  The  reporting  phase  or  the  preparation  of 
the  technical  summary  of  the  entire  test  begins  the  calendar  day-  fol¬ 
lowing  the  completion  of  the  last  test  mission.  No  resources  other 
than  those  required  for  data  reduction,  report  preparation,  and 
deposition  of  test  related  resources  will  be  expended.  The  reporting 
phase  ends  when  the  technical  report  is  signed. 

Suspended  Phase  -  When  a  test  cannot  be  continued  for  a 
significant  period  of  time  the  test  will  be  suspended.  Normally-  a 
teat  will  not  be  placed  in  suspension  unless  a  delay-  of  thirty-  days  is 
anticipated.  This  phase  begins  upon  publication  of  a  letter  of  sus¬ 
pension  and  ends  upon  publication  of  a  letter  placing  the  effort  back 
into  another  phase.  A  suspended  test  may  require  an  amendment  to 
the  test  directive  or  an  amended  Statement  of  Capability  to  reflect 
changes  in  completion  dates,  significant  test  redirection,  and  cost 
changes . 

Closeout  PhaBe  -  The  closeout  phase  begins  upon  activity 
completion  and  ends  sixty  calendar  days  later.  The  purpose  of  this 
phase  is  to  allow  for  the  final  accumulation  of  charges  which  should 


accrue  to  the  test  effort.  After  the  closeout  phase  no  further  charges 
can  be  accepted  under  that  JON. 


In  the  areas  selected  systems  within  the  last  three  years 
that  were  at  least  into  the  reporting  phase  were  examined  for  inclu¬ 
sion  in  the  data  set.  By  including  only  the  most  recently  completed 
systems,  the  effect  of  changing  technology  was  kept  to  a  minimum. 
Radio  Corporation  of  America  (RCA)  has  the  contract  for  collecting 
and  maintaining  the  data  base  for  tests  conducted  by  AD.  No  sus¬ 
pended  programs  were  considered  for  inclusion  because  the  uncer¬ 
tainty  of  scheduling  and  test  objectives  of  a  suspended  system  cause 
unexpected  changes  in  the  total  cost  of  the  system. 

Of  the  final  systems  included,  the  breakout  by  division  are 
outlined  in  Table  1. 


Table  1 

Systems  Included  by  Division 


DIVISION 


NO.  OF  SYSTEMS 


ELECTRONIC  WARFARE  (TZW) 
Attack  (TZWA) 

Defense  (TZWD) 


9 

2 


ELECTRONIC  SYSTEMS  (TZE) 
Airborne  (TZEA) 

TEST  REQUIREMENTS  AND 
PROGRAMMING  (TZP) 
Electronics  (TZPE) 


3 


All  cost  figures  were  collected  in  terms  of  the  fiscal  year 
they  were  charged  and  adjusted  to  constant  1982  dollars.  The  DOD 
raw  inflation  indices  for  RDT&E  were  used  in  the  inflation  adjustment 
process.  The  procedure  for  this  process  is  to  divide  the  actual 
fiscal  year  cost  figure  by  the  index  for  that  year  to  convert  the  fiscal 
year  dollars  into  the  1982  base  year  dollars.  The  index  is  from  Air 
Force  Regulation  (AFR)  173-13,  USAF  Cost  and  Planning  Factors  and 
is  listed  in  Table  2. 


Table  2 


DOD  Raw  Inflation  Indices 
as  of  1  February  1983 


CONVERSION  INDEX 

FISCAL  YEAR  FOR  RDT&E 


1980 

.818 

1981 

.916 

1982 

1.000 

1983 

1.050 

The  results  of  the  data  collection  process  are  shown  in 
Table  3  .  An  expanded  version  of  the  cost  data  appears  in  Appendix  B. 

The  number  of  months  listed  only  includes  the  time  the  pro¬ 
gram  was  in  the  active  phase  of  testing.  The  aircraft  category  was 
determined  by  the  direct  cost  of  a  one  hour  flight  for  the  dominant 
aircraft  in  the  test  program.  The  costs  per  flying  hour  were  obtained 
from  AD's  Standard  Rate  Pricing  Catalog.  These  costs  include  base 
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level  maintenance  and  aviation  petroleum ,  oil,  and  lubricants  (POL) 
(Table  4) . 

Table  4 

AD's  Cost  Per  Flying  Hour 
FY  1983 


AIRCRAFT 


CLASS 


T-38 

$  1026.58 

A- 10 

1487.34 

F- 16 

2649.38 

F-4E 

4103.01 

F-4D 

4551.44 

F-15 

4595.07 

F-4C 

5298.94 

F-lll 

5748.02 

One  exception  was  made  in  categorizing  the  Strike  Shield 
program.  Although  the  dominant  aircraft  was  the  F-15,  over  forty 
percent  of  the  sorties  were  flown  by  an  F-lll.  This  program  was 
put  in  a  higher  class  (4)  because  of  the  relatively  high  cost  of 
flying  the  F-lll. 

The  test  variable  was  assigned  a  value  depending  on  the  type 
of  system  being  tested.  Each  system  was  placed  in  a  general  class 
as  outlined  by  Mr.  Allen  (Table  5). 

Data  Features 


Strengths 


A  strength  of  the  collected  data  is  that  for  each  job  there  is 


Table  5 


General  Class  and  Number  of 
S ys terns  In  Each  Class 


TYPE  OF  SYSTEM  TESTED  CLASS  #  OF  SYSTEMS 

DISPENSERS  1  5 

EXTERNAL  2  3 

INTERNAL  3  _  _ 7 


a  precise  cost  factor  for  resources  used  in  support  of  that  job.  This 
is  evident  by  the  previous  chart  of  cost  per  flying  hour  for  specific 
aircraft.  There  are  also  factors  for  other  resources  such  as  for 
the  use  of  range  radars  or  photographic  equipment.  By  using  the 
Standard  Rate  Pricing  Catalog  the  cost  centers  will  know  exactly  how 
much  to  charge  to  a  specific  JON* 

The  costs  are  then  collected  through  the  JOCAS  to  develop 
the  total  to  be  billed  the  user.  This  cost  collection  system  has  been 
in  effect  since  before  1980  and  all  the  test  data  occurs  after  this  date 
Therefore,  each  system's  costs  were  collected  and  documented  the 
same  way  which  makes  them  easily  comparable.  Also,  since  all  of 
the  systems  were  tested  in  the  last  three  years  the  costs  associated 
with  technology  changes  were  kept  to  a  minimum. 

Other  strengths  of  the  data  were: 

1.  All  the  data  is  objective  in  nature  and  stated  in  quanti¬ 
tative  terms. 
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2.  There  appeared  to  be  no  bias  in  the  data  from  the 


sources  from  which  it  was  gathered. 

3.  All  numerical  conversions  used  established  DOD 
published  rates  (e.g.,  inflation  index). 

Weaknesses 

Program  delays  caused  by  unsuccessful  missions  can  cause 
unexpected  increases  in  the  cost.  After  each  flight  a  Mission 
Accomplishment  Report  (AD  Form  57)  is  submitted  by  the  test  engi¬ 
neer.  This  form  lists  the  objective,  results,  completion  status, 
productivity,  and  reasons  for  failure  or  cancellation.  Some  possible 
reasons  for  an  unsuccessful  mission  include  weather,  test  item  mal¬ 
function,  aircraft  configuration  incompatibility,  aircraft  system 
malfunction,  aircraft  not  available  due  to  maintenance,  lack  of 
technical  support,  and  safety.  Each  day  these  forms  are  submitted 
to  the  technical  advisor  to  the  Commander  for  Test  Engineering  who 
determines  if  the  flight,  all  or  nothing,  should  be  charged  to  the  user. 
If  cancelled  by  weather  or  an  AD  aircraft  or  system  malfunction, 
the  Test  Wing  will  absorb  the  costs.  The  cost  recorded  are  for 
successful  missions  and  missions  where  the  customer  item  (test 


item)  malfunctioned.  The  level  of  delays  or  extra  flights  needed  to 
meet  the  test  objectives  is  absorbed  in  the  total  cost  charged  to  the 
user.  All  costs  charged  to  the  user  are  included  in  the  data  base. 


There  are  no  provisions  distinguishing  between  programs  that  were 
completed  on  schedule  and  those  that  incurred  delays. 


Another  weakness  in  the  data  is  that  some  of  the  programs 
are  still  in  the  reporting  phase.  As  stated  earlier,  no  resources 
other  than  those  required  for  data  reduction,  report  preparation,  and 
deposition  of  test  related  resources  will  be  expended.  All  flying, 
range  operation,  and  engineering  activities,  which  are  the  major  con¬ 
tributors  to  the  costs,  have  been  completed  by  this  time.  Some  com¬ 
puter  support  may  be  required  to  combine  and  analyze  the  data  from 
each  individual  flight.  The  effort  used  to  complete  the  final  report  is 
billed  under  the  test  management  category.  Six  of  the  programs  were 
still  in  the  reporting  phase  with  some  closer  to  completion  than  others, 
The  programs  still  in  reporting  were: 

Countermeasures  Subsystems 
ALR-56  Restructuring 
Improved  Strategic  Chaff 
Strike  Shield 
AN/ALQ  131  RP 
Aerodynamic  Flare  Test 

When  a  program  reaches  this  stage  approximately  90%  of  the  program 


costs  have  been  collected  and  billed  to  the  user. 


CHAPTER  4 


PARAMETRIC  COST  MODEL  DEVELOPMENT 

Logic  of  the  Relationship 

Prior  to  evaluating  the  model  by  statistical  techniques,  it  is 
necessary  to  identify  a  cause  and  effect  relationship  between  the 
selected  independent  and  dependent  variables.  The  development  of  a 
logical  relationship  allows  for  a  guide  for  transforming  variables  as 
well  as  providing  an  additional  check  on  the  validity  of  the  model. 

As  the  first  independent  variable,  number  of  flight  hours, 
increases  the  cost  of  the  flight  test  program  will  increase.  This 
increase  in  cost  should  occur  at  a  constant  rate  as  AD  has  a  factor 
to  charge  the  user  for  each  flying  hour.  Each  hour  flown  will  cost 
the  same  as  the  previous  hour. 

The  second  independent  variable,  number  of  sorties,  follows 
the  same  logic  as  the  flight  hours  since  the  user  should  expect  to  pay 
more  as  the  number  of  sorties  increase.  This  increase,  however, 
should  occur  at  increasing  rate.  The  earlier  flights  are  used  to  set 
up  the  equipment  to  make  sure  everything  is  working  properly.  In 
later  flights  the  actual  testing  will  be  done.  During  these  later  test 
flights,  when  the  actual  mission  is  being  simulated,  more  resources 


will  be  used  which  will  drive  the  cost  up  at  an  increasing  rate. 

As  the  third  independent  variable,  length  of  the  program  in 
months,  increases,  the  cost  will  decrease.  A  program  that  has  a 
higher  priority  and  takes  a  shorter  time  to  complete  may  overwork 
the  system.  Additional  resources,  overtime  salaries,  and  extra 
maintenance  maybe  needed  to  meet  the  requirements  of  the  acceler¬ 
ated  program.  Assuming  the  test  is  a  significant  size  for  the  RTO 
the  shorter  time  may  have  adverse  effects  of  the  cost  to  complete 
the  test.  For  longer  test  programs,  that  do  not  place  special 
requirements  on  the  existing  resources,  the  test  will  run  more 
efficiently  and  cost  less  to  accomplish.  This  decrease  will  occur 
at  a  linear  rate  over  the  relevant  range  of  the  data.  Finally  as  the 
program  continues  to  drag  on  the  cost  is  expected  to  increase. 

The  remaining  independent  variables  are  indicator  or  dummy 
variables  that  classify  the  test  into  specific  categories.  Dummy 
variables  are  used  to  account  for  the  fact  that  the  observations  within 
a  given  category  are  associated  with  one  set  of  regression  para¬ 
meters,  while  observations  in  a  second  (or  third)  category  are  asso¬ 
ciated  with  a  different  set  of  regression  parameters.  The  type  of 
aircraft  will  effect  the  cost  as  some  are  more  expensive  to  fly  and 
maintain  than  others.  The  final  independent  variable  is  the  classifi¬ 
cation  of  the  type  of  testing  being  done.  By  distinguishing  between 
tests,  systems  that  have  similar  effects  on  costs  can  be  grouped 


together. 


Potential  Model  Structures 

The  sample  consisted  of  the  fifteen  electronic  system  flight 
test  programs.  The  costs  were  assumed  to  be  a  function  of  the  fol¬ 
lowing  independent  variables: 

#  Sorties 

#  Months 

#  Flight  Hours 
A,  B,  C 

X,  Y 

where  A,  B  and  C  represent  the  dummy  variable  for  the  type  of 
aircraft  and  X,  Y  represent  the  dummy  variable  for  the  type  of  test. 
In  using  the  dummy  variables,  whose  values  are  either  0  or  1,  the 
test  aircraft  and  type  of  system  being  tested  must  be  placed  into  a 
class.  In  the  case  of  test  aircraft  there  are  four  possible  classes. 
Depending  on  the  class,  the  coefficient  is  multiplied  by  either  0  or  1 
to  include  the  effects  of  that  variable  on  the  dependent  variable.  The 
values  of  the  variables  that  correspond  with  each  class  are  given  in 
Table  6  . 
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Table  6 


i 


Variable  Values  for  Each 
Aircraft  Class 


CLASS  VARIABLE 

ABC 


1  10  0 

2  0  10 

3  0  0  1 

4  0  0  0 


As  an  example  if  the  test  aircraft  is  an  F-16  (Class  2),  a  1  would  be 
inserted  in  the  equation  for  variable  B  while  the  A  and  C  would  take 
on  the  value  of  0 . 

In  tiie  case  of  the  type  of  testing,  the  corresponding  values 
for  the  X  and  Y  variables  are  given  in  Table  7 . 

I 

|  % 

Table  7 

1 

|  Variable  Values  for  Each 

I  Test  Class 


CLASS 


VARIABLE 
X  Y 


To  assign  the  best  specification  whose  behavior  was  con¬ 


sistent  with  prior  expectations  the  data  base  was  input  into  the  SPSS 
program  and  run  as  a  linear  function.  The  initial  run  was  analyzed 
to  look  for  specification  errors  by  the  use  of  net  scatter  diagrams. 

To  draw  a  net  scatter  diagram  the  value  of  the  independent 
variable  is  plotted  on  the  X  axis.  The  net  regression  line  for  that 
variable  is  graphed  (slope  =  b  value  for  the  specific  variable  from 
the  linear  run).  Next  the  residual  for  each  of  the  observations  is 
plotted  as  the  distance  from  the  net  regression  line.  The  plot  of 
residuals  should  appear  randomly  distributed  about  the  regression 
line  if  the  specification  is  correct.  The  graphs  serve  as  a  basis  for 
decisions  as  to  whether  transformations  are  called  for  and  what 
transformations  should  be  considered. 

The  net  scatter  diagram  for  the  independent  variables  flight 
hours  and  months  had  no  apparent  patterns.  This  suggests  that  these 
variables  are  properly  specified  which  is  consistent  with  the  previous 
established  logic.  The  graph  for  the  variable  number  of  sorties  had 
a  pattern  suggesting  a  non-linear  relationship. 

Collinearity 

A  problem  that  resulted  from  this  run  was  that  the  coefficient 
for  the  number  of  flight  hours  was  negative.  This  means  the  first 
derivative  of  this  variable  is  negative  which  violates  the  logic  or 


37 


expected  behavior  for  this  variable.  This  result  can  be  caused  by 
the  relatively  small  size  of  the  sample  as  there  is  always  a  chance 
of  getting  unexpected  outcomes.  Another  explanation  for  this  problem 
is  the  existence  of  collinearity  between  this  variable  and  the  number 
of  sorties. 

Collinearity  occurs  when  there  exists  a  relationship  between 
two  or  more  independent  variables.  In  a  regression  model,  collin¬ 
earity  causes  instability  in  the  results  as  analysis  produces  large 
variances  which  increases  the  confidence  interval  around  the 
regression  line.  By  looking  at  the  correlation  matrix,  which  looks 
at  the  pairwise  relationship  between  variables,  it  is  evident  by  the 
value  of  .998  that  the  variables  flight  hours  and  sorties  are  very 
strongly  related.  The  reason  for  this  is  that  the  sample  consists 
mainly  of  fighter  aircraft  and  for  these  types  of  aircraft  the  sortie 
length  normally  falls  between  one  and  two  hours.  Each  mission 
flown  will  add  approximately  the  same  number  of  flight  hours  to  the 
total  of  the  test  flight  hours. 

To  reduce  this  collinearity,  other  than  increasing  the  sample 
size,  the  related  variables  can  be  combined  to  form  a  new  variable 
or  one  of  the  variables  can  be  eliminated.  By  dividing  flight  hours 
by  sorties  a  new  variable,  defined  as  average  sortie  length,  was 
created  and  included  as  one  of  the  independent  variables.  The  num¬ 
ber  of  sorties  remained  in  the  equation  while  the  number  of  flight 
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hours  was  eliminated.  This  particular  combination  was  chosen 
because  it  still  measures  the  same  theoretical  characteristics  (flight 
hours)  and  is  related  to  costs.  This  new  variable  would  have  a  posi¬ 
tive  effect  on  costs  as  the  length  of  the  sortie  increased  the  cost 
should  increase.  This  increase,  however,  should  occur  at  a  linear 
rate  because  of  the  set  flying  hour  costs. 

Average  sortie  length  was  included  in  the  model  and  the 

equation  was  again  run  as  a  linear  function.  The  new  variable  and 

number  of  sorties  were  not  highly  correlated  as  the  value  from  the 

correlation  matrix  was  -.218.  An  additional  test  for  collinearity  that 

goes  beyond  pairwise  correlation  is  the  tolerance  test.  Tolerance  is 
2  2 

defined  as  1  -  R.  where  Rj  is  the  coefficient  of  determination  when 
one  independent  variable  is  regressed  against  the  remaining  inde¬ 
pendent  variables.  A  high  tolerance  value  (close  to  one)  means  a  low 
correlation.  The  tolerance  values  were  in  Table  8. 

Table  8 

Tolerance  Levels  for  Each  Variable 


.777 

.425 


SORTIE 

AVERAGE  SORTIE  LENGTH 


MONTHS 


.723 


Generally,  if  one  or  more  of  these  values  is  approximately  the  same 
size  as  the  R  from  the  regression  equation  then  multicollinearity  is 
a  problem.  In  other  words,  if  the  strength  of  the  association  among 
any  of  the  independent  variables  is  approximately  as  great  as  the 
strength  of  their  combined  linear  association  with  the  dependent 
variable,  then  the  amount  of  overlapping  influence  may  be  substantial 
enough  to  make  the  interpretation  of  the  results  difficult  and  impre¬ 
cise  (10:563).  In  the  worst  case  57.5  percent  of  the  value  of  average 
sortie  length  can  be  explained  using  the  remaining  variable  (as  com¬ 
pared  to  over  .80  for  the  linear  run  with  the  dependent  variable).  By 
creating  the  new  variable  average  sortie  length  the  collinearity  in  the 
model  was  reduced  and  no  longer  was  causing  adverse  effects  on  the 
model. 

Outliers 

The  data  set  was  analyzed  for  outliers  and  to  determine  if 
these  outliers  were  influential.  The  computer  printout  listed  the 
studentized  residual  values  for  the  data  points  that  were  potential 
outliers.  These  values  are  compared  to  the  t  distribution  value  with 
6(n-k)  degrees  of  freedom  and  a  confidence  level  of  .90.  This  con¬ 
fidence  level  was  chosen  as  it  is  a  standard  level  used  in  statistical 
analysis.  This  procedure  revealed  that  the  first  and  third  data  points 
entered  were  outliers.  The  way  to  determine  if  these  outliers  are 
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influential  is  with  "Cook's  Distance"  which  is  also  printed  on  the  com¬ 
puter  output.  This  method  compares  values  with  critical  values  of 
the  F  distribution  with  k  and  n-k  degrees  of  freedom.  If  the  calcu¬ 
lated  value  is  in  the  tail,  it  is  considered  to  be  influential.  At  the 
90%  level  of  confidence  the  critical  value  was  2.55  which  was  greater 
than  the  calculated  value  from  both  the  first  and  third  data  points. 
Therefore,  although  there  were  outliers  they  were  not  influential  at 
the  90%  level. 

The  observations  that  were  identified  as  outliers  included 
the  Countermeasures  Subsystems  program  and  the  Electronic 
Countermeasure  (ECM)  Trial  Phase  Two  program.  They  were 
similar  systems  in  that  they  flew  the  most  sorties,  89  and  73 
respectively,  in  a  relatively  short  period  of  time.  Both  programs 
had  at  least  6%  of  the  total  cost  listed  as  miscellaneous  expenses 
that  are  not  evident  on  the  other  programs.  This  expense  which 
amounts  to  over  $156,000  in  the  Countermeasure  Subsystem  and 
$86,000  for  the  ECM  Trial  Phase  Two,  is  a  probable  source  in 
making  these  points  outliers.  There  is  no  evidence  of  an  error  in 
recording,  instrumentation  malfunctioning  or  observer  measure¬ 
ment  mistakes.  Although  outliers  exist  there  is  no  reasonable  and 
consistent  justification  for  their  exclusion  ami  at  the  90%  level  they 
tested  to  be  non- influential  outliers. 

After  checking  for  collinearity  and  outliers,  various 


statistical  tests  were  performed  with  the  linear  run  to  determine  the 
degree  to  which  the  independent  variables  predict  the  dependent 
variable.  The  first  test  was  the  coefficient  of  determination  (R  ) 
which  measures  the  portion  of  the  dependent  variable  measured  by 
the  regression  model.  The  for  the  linear  run  was  .83  which 
exceeded  the  accepted  level  of  .80. 

The  F  test,  which  tests  the  null  hypothesis  that  there  is  no 
linear  relationship  at  all  in  the  population  (i.e.,  all  the  B  values 
equal  zero)  was  set  at  the  .80  level  in  order  for  the  model  to  be 
considered  statistically  significant.  The  computed  F  value  was  3.78 
which  was  greater  than  the  critical  value  (with  eight  and  six  degrees 
of  freedom).  Therefore,  the  null  hypothesis  was  rejected  and  the 
hypothesis  that  the  equation  is  useful  in  describing  costs  was  accepted 
at  the  80  percent  level. 

The  t  test  was  used  to  test  the  significance  of  the  individual 
coefficients  in  the  regression  model.  Computed  "t"  values  were 
compared  to  critical  "t"  values  at  the  .80  level  of  confidence.  The 
most  significant  variable  was  the  number  of  sorties  at  a  level  of 
.9966.  The  number  of  months  also  met  the  criteria  as  it  was 
significant  at  a  level  of  .8356.  Ihe  created  variable,  average  sortie 
length,  did  not  meet  the  set  criteria  and  was  eliminated  from  the 
model.  It  did  little  to  explain  the  dependent  variable  because  the 
data  set  included  aircraft  with  little  variation  in  the  length  of  the 


sorties.  The  remaining  t  values  for  the  individual  coefficients  that 
made  up  the  dummy  variables  were  not  significant  but  this  has  little 
meaning  for  the  model.  All  the  variables  (value  either  zero  or  one) 
are  used  to  place  the  aircraft  and  test  into  the  proper  category. 

These  single  variables  have  little  meaning  when  they  are  alone  but 
need  to  be  taken  as  a  set  to  add  meaning  to  the  model. 

With  the  variable  average  sortie  length  eliminated,  the 
equation  was  again  run  as  a  linear  function.  Net  scatter  diagrams 
were  drawn  for  the  variables,  number  of  sorties  and  length  in  months. 
They  are  shown  in  Figure  1  and  Figure  2,  respectively.  The  length 
in  months  appeared  properly  specified  while  the  diagram  for  number 
of  sorties  suggested  a  non-linear  function.  Therefore,  to  improve  on 
the  specification  of  the  model  the  number  of  sorties  was  transformed 
to  a  squared  variable.  A  full  quadratic  function  (SORTIE2  +  SORTIE) 
was  not  used  because  of  the  limited  number  of  tests  in  the  sample. 
Also,  this  combination  was  run  and  the  coefficient  for  the  variable 
number  of  sorties  did  not  meet  the  previous  set  criteria,  while  the 
coefficient  for  number  of  sorties2  remained  significant. 

Evaluating  the  Selected  Model 

In  evaluating  the  validity  of  the  model  emphasis  was  given  to 
the  principle  that  the  specific  functional  relationship  between  inde¬ 
pendent  variables  (cost  drivers)  and  the  dependent  variable  (cost) 


Scatter  Diagram  for  Number  of  Sorties 


Net  Scatter  Diagram  for  Number  of  Months 


be  consistent  with  previous  expectations.  After  making  the  trans¬ 
formation  the  signs  of  the  coefficients  were  checked  to  see  if  they 
would  produce  the  proper  first  and  second  derivatives  for  that 
particular  variable.  Without  the  underlying  logic  the  CER  can  be 
misleading. 

The  independent  variable,  number  of  sorties,  was  squared 
and  the  coefficient  was  positive.  This  is  consistent  with  the 
expected  outcome  as  the  cost  is  increasing  at  an  increasing  rate 
with  the  number  of  sorties.  The  variable,  number  of  months,  was 
left  as  a  linear  function  and  had  a  negative  coefficient. 

The  R  value  increased  from  the  linear  from  .834  to  .881 
as  more  of  the  variation  was  explained.  The  F  value  increased  to 
7.42  with  a  significance  level  of  .9916  which  surpasses  the  set 
criteria.  The  number  of  sorties  was  the  most  significant  variable 
with  the  length  in  months  being  the  next  most  significant. 

The  collinearity  was  at  a  level  where  it  was  not  having  an 
adverse  effect  on  the  model  as  the  highest  value  in  the  correlation 
matrix  was  .645  which  occurred  between  two  of  the  dummy  variables. 
The  tolerance  value  for  number  of  sorties  and  months  were  .83822 
and  .75722,  respectively.  These  high  tolerance  values  mean  a  low 
correlation  when  one  independent  variable  is  regressed  against  the 
remaining  independent  variables.  The  first  data  point  was  a  possible 
influential  outlier  but  there  was  no  reasonable  and  consistent 


justification  for  eliminating  it. 

In  addition  to  the  statistical  tests  a  relative  error  cal¬ 
culation  was  made.  Table  9  shows  the  relative  error  for  each  point 
estimate.  This  relative  error  calculation  can  be  used  by  the  decision 
maker  to  access  the  risk  in  accepting  the  estimate. 

The  results  of  this  final  run  are  included  in  Appendix  C. 

In  a  final  run  an  additional  test  for  the  proper  specifications  is  the 
standardized  scatterplot  of  the  independent  variables  and  the 
residual.  If  the  specification  is  correct  the  residuals  should  be 
randomly  distributed  in  a  band  about  the  horizontal  straight  line 
through  zero.  From  these  scatterplots  both  variables  appear  to  be 
properly  specified. 

Based  on  the  results  of  the  logic  established  and  the  trans¬ 
formation  performed  the  selected  model  was: 

Y  =  307.1919  +  .23642  <#  Sorties)2  -  17.0513  (#  Months) 

+  115.54565(A)  -  124.83783(B)  -  16.39536(C) 

+  178.66428  (X)  +  56.23335(Y)  +  e. 


Actual  Coat  Va.  Point  Eatimate  and  Relative  Error 
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CHAPTER  5 


MODEL  USAGE  AND  SUMMARY 


The  CER  developed  by  this  thesis  effort  may  be  used  to 
predict  flight  test  costs  for  electronic  systems.  The  cost  was  defined 
as  the  direct  cost  charged  to  the  SPO  for  the  RTO  to  complete  the 
test.  This  model  can  be  used  by  the  SPO  to  make  budget  estimates 
early  in  the  program's  life.  Estimates  from  this  model  will  be  in 
FY82  dollars,  therefore,  care  should  be  taken  in  converting  the 
FY82  dollars  to  the  appropriate  current  year  dollars. 

The  CER  is  used  by  entering  values  of  the  flight  test  char¬ 
acteristics  for  the  system  to  be  estimated  into  the  model.  The 
accuracy  of  the  model  will  be  partially  dependent  upon  the  accuracy 


a 


of  the  data  values  used  in  estimating  the  cost  of  the  system.  Values 
entering  the  model  should  be  checked  for  consistency  and  reason¬ 


ableness  in  relation  to  existing  systems  in  the  data  base. 


i, 


An  area  of  risk  in  the  accuracy  of  the  variable  inputs  is  the 
possibility  of  cancelled  or  unsuccessful  missions. 


A 

v! 


Eglin  program  engineers,  who  are  in  charge  of  making 
schedule  and  cost  estimates,  make  provisions  for  such  unexpected 
events.  If  ten  successful  missions  are  needed  to  complete  the  test 


the  programmer  normally  will  schedule  fifteen  missions.  This  1.5:1 
ratio  is  the  guideline  for  testing  of  systems  where  similar  tests 
have  previously  been  successful.  As  an  example,  this  ratio  would 
apply  when  a  system  was  moved  from  one  aircraft  to  another  or  when 
a  system  was  updated  and  improved.  In  the  event  of  an  increase  in 
the  state-of-the-art,  this  ratio  would  increase.  In  the  event  of  tech¬ 
nological  change  twice  as  many  sorties  may  be  scheduled,  and  if  the 
uncertainty  is  very  high,  the  programmer  may  decide  to  move  this 
ratio  to  as  high  as  5:1.  As  the  uncertainty  in  die  system  and  the 
means  of  testing  the  system  is  reduced,  the  percentage  of  successful 
flights  will  increase.  Once  the  number  of  successful  sorties  needed 
to  complete  the  test  is  determined  a  factor  should  be  applied  to 
reduce  the  risk  of  predicting  the  cost. 

Summary 

The  objective  of  this  research  was  to  develop  a  cost  pre¬ 
diction  model  for  electronic  system  flight  test  costs.  Answers  to 
the  research  questions  provided  the  means  to  fulfill  the  research 
objective.  The  costs  included  in  flight  testing  are  categorized  and 
billed  to  the  customer.  These  costs  are  outlined  by  program  in 
Appendix  B.  The  significant  cost  drivers  in  developing  an  estimation 
model  for  flight  test  include  the  number  of  sorties  flown,  the  length 
of  the  program,  the  length  of  each  sortie,  the  aircraft  used  in  the 


test,  and  the  type  of  system  being  tested.  These  cost  drivers  were 
combined  through  logic  and  statistical  techniques  to  form  an  equation 
to  predict  future  flight  test  costs. 


The  research  and  the  CER  developed  from  this  thesis  effort 
has  been  done  primarily  to  enhance  and  to  aid  the  cost  estimating 
capabilities  of  the  Program  Control  Division  of  the  Reconnaissance 
and  Electronic  Warfare  System  Program  Office,  Aeronautical 
Systems  Division,  Air  Force  Systems  Command.  The  work  pre¬ 
sented  relies  on  the  assumption  that  the  CER  could  be  developed 
based  on  the  flight  test  characteristics.  The  model  developed  is 
applicable  to  those  systems  which  are  from  the  same  technological 
base  as  the  systems  in  the  current  data  base.  To  keep  the  model 
current,  new  data  should  be  collected  and  added  as  it  becomes  avail¬ 
able.  As  changes  are  made  to  the  existing  data,  the  coefficients  of 
the  variables  and  the  variables  themselves  may  change.  Only  with 
a  continual  maintenance  and  upkeep  of  the  data  base  will  the  CER  be 
able  to  be  used  as  an  effective  tool  for  estimating  electronic  systems 
flight  test  costs  over  any  period  of  time. 

Limitations  of  the  Model 

The  dependent  variable  in  the  model  was  defined  as  direct 
costs  and  does  not  estimate  the  total  cost  of  the  flight  test  effort. 
Also,  there  are  no  provisions  to  examine  the  impact  on  cost  of  test 


item  malfunctions  or  program  delays. 

Another  limitation  of  the  model  is  that  the  coefficient  for 
variable  number  of  months  was  negative.  Although  reasons  were 
given  for  this  outcome  it  is  not  intuitively  appealing  in  the  general 
case.  This  may  be  caused  by  an  inherent  problem  in  the  data  base. 
Due  to  the  time  constraint  to  complete  this  study  this  problem  was 
not  further  examined. 
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APPENDIX  A 

ORGANIZATIONAL  CHART 
OF  3246 TH  TEST  WING 
EGLIN  AFB  FLORIDA 


APPENDIX  B 


COUNTERMEASURES  SUBSYSTEM 
JON  5615WA07 


hi 


FY  1981 

FY  1982 

Aircraft  and  Logistics 

104 

427991 

Computer  Support 

189 

118397 

Engineering  Support 

7216 

976017 

Range  Operation 

— 

255711 

Range  Support 

— 

14 100 

Test  Management 

2522 

58720 

Test  Requirements  Support 

8879 

4229 

Aircrew  Support 

---- 

954 

Airborne  Support 

---- 

58 

Photographies 

---- 

7299 

Electronics 

---- 

— 

Miscellaneous 

751 

140000 

FY  1983 
207924 
93351 
22656 
115538 
399 
16989 

37154 

2647 

22602 

16251 
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STRIKE  SHIELD 
JON  268 3 WA 11 


FY  1981 

FY  1982 

FY  1983 

Aircraft  and  Logistics 

---- 

362 

457614 

Computer  Support 

624 

17332 

105516 

Engineering  Support 

10918 

165223 

7139 

Range  Operation 

764 

63031 

1 148 12 

Range  Support 

— 

411 

2113 

Test  Management 

76 

503 

7329 

Test  Requirements  Support 

1275 

6649 

---- 

Aircrew  Support 

— 

— - 

4026 

Airborne  Support 

---- 

---- 

66867 

Photographies 

«•  m  •  *■ 

69 

1228 

Electronics 

---- 

10775 

Miscellaneous 

---- 

631 

481 

Total 

13657 

254215 

777900 

FEATURED  ECM  TRIAL  PHASE  TWO 
JON  2683WA12 


*b 

r  4 
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E 


FY  1982 

FY  1983 

Aircraft  and  Logistics 

546023 

---- 

Computer  Support 

99217 

41156 

Engineering  Support 

65636 

---- 

Range  Operation 

60639 

1446 

Range  Support 

122 

---- 

Test  Management 

10677 

10493 

Test  Requirements  Support 

13270 

Aircrew  Support 

1269 

---- 

Airborne  Support 

277 

Photographies 

122 

33 

Electronics 

---- 

— 

Miscellaneous 

86522 

360 

Total 

883497 

53765 
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F-16  ALR-69  IMPROVEMENT 
JON  ASD0WA45 


FY  1981 

FY  1982 

FY  1983 

Aircraft  and  Logistics 

27225 

190434 

47155 

Computer  Support 

17288 

69574 

19101 

Engineering  Support 

45447 

44060 

6797 

Range  Operation 

49232 

223761 

59085 

Range  Support 

34 

374 

---- 

Test  Management 

209 

2479 

— 

Test  Requirements  Support 

3825 

2537 

---- 

Aircrew  Support 

—  - 

---- 

9138 

Airborne  Support 

m  aa  •  • 

---- 

10101 

Photographies 

---- 

276 

---- 

Electronics 

•  •  «•  aa 

m  •  «•  M 

Miscellaneous 

W  M  «  « 

Total 

143260 

533495 

151377 
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£LR-56  restructuring 
JON  5618WA11 


FY  1981 

FY  1982 

FY  1983 

Aircraft  and  Logistics 

238383 

67376 

23894 

Computer  Support 

49669 

46552 

9612 

Engineering  Support 

17721 

1622 

---- 

Range  Operation 

238332 

60847 

15605 

Range  Support 

1790 

---- 

---- 

Test  Management 

---- 

5594 

1085 

Test  Requirements  Support 

1491 

---- 

---- 

Aircrew  Support 

171 

— 

Airborne  Support 

•  ■»  •  mm 

—  - 

.... 

Photographies 

•  •  • 

---- 

---- 

Electronics 

---- 

---- 

B  W  M  ■» 

Miscellaneous 

-  -  -  - 

---- 

Total 


547557 


181991 


47805 


ALiQ  119  LENS  ARRAY  DT&E 
JON  921AWF21 


FY  1980 

FY  1981 

Aircraft  and  Logistics 

86024 

146946 

Computer  Support 

43712 

20761 

Engineering  Support 

3162 

4074 

Range  Operation 

80014 

46  776 

Range  Support 

---- 

89 

Test  Management 

27299 

6294 

Test  Requirements  Support 

— 

428 

Aircrew  Support 

---- 

---- 

Airborne  Support 

— - 

---- 

Photographies 

170 

Electronics 

«■  m  m  mt 

Miscellaneous 

— 

---- 

Total 

240211 

225598 

62 


Aircraft  and  Logistics 
Computer  Support 
Engineering  Support 
Range  Operation 
Range  Support 
Test  Management 
Test  Requirements  Support 
Aircrew  Support 
|  Airborne  Support 

Photog  raphic  s 
Electronics 
Miscellaneous 


AN/ALQ  131  RP 
JON  2272WA03 


Y  1981 

FY  1982 

FY  1983 

---- 

116445 

49848 

312 

39113 

11367 

7176 

---- 

96 

109092 

46516 

4497 

28730 

39313 

1536 

4298 

---- 

357 

339 

73 

Total 


6441 


305193 


147474 


A- 10  ALR-69  IMPROVEMENT 
JON  ASD0WA47 


FY  1981 

FY  1982 

FY  1983 

Aircraft  and  Logistics 

7811 

129527 

10 10 

Computer  Support 

2572 

29596 

10060 

Engineering  Support 

17731 

47873 

507 

Range  Operation 

9353 

112595 

39772 

Range  Support 

---- 

3325 

---- 

Test  Management 

1520 

703 

6238 

Test  Requirements  Support 

2462 

3111 

Aircrew  Support 

m  •»  m  mm 

---- 

Airborne  Support 

ae  m  •  • 

—  — 

5616 

Photographies 

---- 

725 

— * 

Electronics 

---- 

««  M  M  M 

mm  w  m  m 

Miscellaneous 

924 

---- 

Total 


42373 


324344 


66314 


PEACE  FOX 
JON  FDISWAO 1 


FY  1981 

FY  1982 

FY  1983 

Aircraft  and  Logistics 

29777 

72856 

---- 

Computer  Support 

10794 

22477 

8887 

Engineering  Support 

364 

2138 

10784 

Range  Operation 

54072 

96992 

414 

Range  Support 

---- 

---- 

— 

Test  Management 

729 

10987 

3934 

Test  Requirements  Support 

234 

313 

---- 

Aircrew  Support 

435 

356 

Airborne  Support 

.... 

---- 

Photographies 

•  •  •  M 

— - 

Electronics 

---- 

•  •  •  m 

Miscellaneous 

---- 

.... 

•m  m  mt  m 

Total 


96405 


206119 


240 19 


F-16  RAM  TEST 
JON  ASD0WA37 


FY  1980 

FY  1981 

Aircraft  and  Logistics 

8858 

76683 

Computer  Support 

9014 

42167 

Engineering  Support 

26889 

18432 

Range  Operation 

105 

82999 

Range  Support 

920 

Test  Management 

944 

4831 

Test  Requirements  Support 

---- 

556 

Aircrew  Support 

136 

Airborne  Support 

— — 

— — 

photographies 

---- 

Electronics 

Miscellaneous 

---- 

— — 

Total 

458 10 

226724 

66 


IMPROVED  STRATEGIC  CHAFF 
JON  2683WA06 


FY  1981 

FY  1982 

FY  1983 

Aircraft  and  Logistics 

4003 

76788 

36210 

Computer  Support 

199 

147 

14381 

Engineering  Support 

---- 

287 

---- 

Range  Operation 

1171 

5276 

22160 

Range  Support 

---- 

---- 

— 

Test  Management 

5544 

6134 

6055 

Test  Requirements  Support 

2637 

2887 

— 

Aircrew  Support 

331 

2734 

— 

Airborne  Support 

tm  m 

509 

Photographies 

---- 

44 

---- 

Electronics 

---- 

---- 

— 

Miscellaneous 

—  w 

aaaM 

aaaa 

Total 


13885 


94297 


79315 
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PEACE  SUN 
JON  FDSRWAO 1 


FY  1981 

FY  1982 

FY  1983 

Aircraft  and  Logistics 

17437 

21087 

---- 

Computer  Support 

639 

6117 

— 

Engineering  Support 

660 

647 

166 12 

Range  Operation 

39428 

59564 

---- 

Range  Support 

---- 

---- 

---- 

Test  Management 

1852 

5424 

14948 

Test  Requirements  Support 

271 

204 

— - 

Aircrew  Support 

194 

139 

m  m  m  m 

Airborne  Support 

m  m  m  m 

m  m  m  m 

Photographies 

.... 

— - 

—  - 

Electronics 

---- 

m  m  m  m 

m  mm  m 

Miscellaneous 

mm  mm 

.... 

---- 

Total 


60481 


93182 


31560 


PEACE  MARBLE 
JON  FMS0WA14 


Aircraft  and  Logistics 
Computer  Support 
Engineering  Support 
Range  Operation 
Range  Support 
Test  Management 
Test  Requirements  Support 
Aircrew  Support 
Airborne  Support 
Photographies 
Electronics 
Miscellaneous 


FY  1982 
34636 
27867 
2485 
106917 

2398 

3792 

272 


Total 


178367 


RR- 180  DUAL  CHAFF  CARTRIDGE 
JON  2274WA04 


FY  1982 

FY  1983 

Aircraft  and  Logistics 

100390 

126 

Computer  Support 

2570 

1744 

Engineering  Support 

12572 

1232 

Range  Operation 

- --- 

— 

Range  Support 

— 

---- 

Test  Management 

10634 

4225 

Test  Requirements  Support 

7130 

---- 

Aircrew  Support 

912 

---- 

Airborne  Support 

— - 

5482 

Photographies 

798 

— 

Electronics 

— 

1744 

Miscellaneous. 

2595 

•  ... 

Total 


137601 


14553 


AERODYNAMIC  FLARE  TEST 
JON  2000EQ12 


FY  1981 

FY  1982 

FY  1983 

Aircrew  and  Logistics 

— 

5138 

56186 

Computer  Support 

— 

820 

8318 

Engineering  Support 

— 

6548 

436 

Range  Operation 

— 

1253 

8581 

Range  Support 

— 

30 

— 

Test  Management 

493 

8004 

8429 

Test  Requirements  Support 

624 

4330 

19928 

Aircrew  Support 

— 

75 

124 

Airborne  Support 

— 

— 

1181 

Photographies 

— 

738 

4019 

Electronics 

— 

— 

— 

Miscellaneous 

—  —  — 

... 

Total 


1117 


26936 


107814 


APPENDIX  C 


RESULTS  OF  THE  FINAL 
COMPUTER  RUN 


1  RUN  NAME 

2  VARIABLES  LIST 

3  INPUT  FORMAT 

4  N  OF  CASES 

5  COMPUTE 

6  NEW  REGRESSION 

7 

8 
9 

10 

11 

12 

13  READ  INPUT  DATA 


FLIGHT  TEST  ANALYSIS 

COST,  FLTHRS,  SRT,  MONTHS,  A,  B,C,X,  Y 

FREEFIELD 

15 

SRT=SRT**2 

DESCRIPTIVES/ 

VARIABLES=COST ,  SRT ,  MONTHS,A,B,C,X,Y/ 
CRITERIA=PIN(0 . 9) ,  POUT(0 . 9)7 
DEPENDENT=COST/ 

STEPWISE/ 

SCATTERPLOTS(*RESID,SRT)/ 
SCATTERPLOTS(*RESID,  MONTHS)/ 


VARIABLE  LIST 


MEAN 

COST 

602.667 

SRT 

1968.133 

MONTHS 

11.400 

A 

0.067 

B 

0.267 

C 

0.533 

X 

0.267 

Y 

0.200 

STD  DEV  LABEL 

614.335 

2375.939 

6.423 

0.258 

0.458 

0.516 

0.458 

0.414 


N  OF  CASES  =  15 
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VARIABLES  IN  THE  EQUATION 


VARIABLE 

Y 

MONTHS 

A 

SRT 


(CONSTANT) 


B 

SE  B 

56.23335 

213.  17950 

-17.05130 

14.31675 

115.54565 

382.62001 

0.23642 

0.03679 

178.66428 

195.95041 

-124.83783 

265.94568 

-16.39536 

265.49237 

307. 19190 

260.91790 
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T  SIGT 
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-0. 17828 

-1.  191  0.2725 

0.04856 
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0.91434 

6.427  0.0004 

0. 13312 

0.912  0.3922 

-0.09302 

-0.469  0.6530 

-0.01378 

-0.062  0.9525 
1. 177  0.2775 
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